#
# fusion.db.db
#
# Copyright 2003-2008 Helsinki Institute for Information Technology
# and the authors.
#
# Authors: Ken Rimey <rimey@hiit.fi>
#

# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

# This file derives from db.py in PDIS (http://pdis.hiit.fi/).

# We require APSW:
#
#   http://initd.org/pub/software/pysqlite/apsw/3.3.13-r1/apsw.html#Building
#
# Migrating to the pysqlite included in Python 2.5 would simplify
# installation, but it has subtle differences, so this is not
# something to rush into.

"""
Sqlite3-based store for collections of signed key-value pairs

This is the main Python API for application developers.  The following
is a quick overview of the methods of a DB instance.

First of all, creating an instance requires specifying the location of
the database on disk:

    from fusion.db.db import DB
    db = DB(path_to_database=':memory:')

The database is created if it does not already exist.

Closing the database before you exit is a good idea:

    db.close()

DATASETS

The main content of the database is organized as shared datasets.
Each dataset has a unique information ID (IID), which is a string of
40 hexadecimal digits.

There are three kinds of datasets: personal profiles, group profiles,
and content datasets.  A personal profile represents a person or
on-line persona, and its IID is the person's IID.  Similarly, a group
profile represents a group of people as defined by a membership list,
and its IID is the group's IID.  Every content dataset is permanently
associated with some person or group.  A personal or group profile,
with its associated content datasets, is called a domain, and the IID
of the profile also serves as the domain's IID.

Each database is normally imprinted with the persona of its owner.  At
most limited functionality will be available if it is not.

After creating a new database, a first time user would normally call
create_persona() to create a new personal profile and imprint the
database with it:

    db.create_persona(name) -> IID
    db.get_my_iid() -> IID
    db.get_persona() -> (IID, master secret)

The name can be changed freely later.  It is stored in the profile as
the value of "/public/name.txt".

A user who already has a personal profile (created at some other
computer) can instead imprint the database with it:

    db.set_persona(iid, secret)
    db.clear_persona()

The user's master secret is sensitive and cannot be reset, so
transporting it safely between computers is important.  Our solution
is to store an encrypted copy, protected by both a computer-generated
code to be written down and a user-selected passphrase to be
remembered.  The user can then fetch the master secret and imprint the
database with the persona by providing both the code and the
passphrase.

    db.reset_personal_code(passphrase) -> string
    db.set_persona_from_code(code, passphrase)

After the database has been imprinted, it is possible to create new
groups or content datasets:

    db.create_group(name) -> IID
    db.create_content(domain, title) -> IID

Creating a content dataset requires specifying the IID of the person
or group in whose domain it is to be included.

Naturally, one can query the basic attributes of a dataset:

    db.get_type(iid) -> 'person', 'group', or 'content'
    db.get_domain(iid) -> IID

These methods return None if the answer is not known.

One can also query what datasets are present in the database:

    db.list_persons() -> list of IIDs
    db.list_groups() -> list of IIDs
    db.list_content(domain) -> list of IIDs
    db.list_subscriptions() -> list of IIDs

The list_subscriptions() method lists all datasets of all three types.
It also lists datasets for which the type is not known, which is
possible if the IID of a dataset that was created elsewhere has been
added to the database.  This is called "subscribing" to a dataset.  It
causes the synchronization agent to try to fetch the contents of the
dataset via the network.

    db.subscribe(iid)
    db.unsubscribe(iid)

Unsubscribing to a content dataset deletes it from the database.
Unsubscribing to a person or group IID removes the entire domain
(profile and associated content datasets) from the database.

Subscribing to a content dataset signals a willingness to subscribe to
the associated personal or group profile.  The type and domain of the
content dataset become known as soon as its data starts arriving, and
a subscription for the domain's profile is automatically entered at
that time.

>>> db = DB()
>>> p = db.create_persona('Test Person')
>>> g = db.create_group('Test Group')
>>> c = db.create_content(p, 'Personal Content')
>>> c2 = db.create_content(g, 'Group Content')
>>> c3 = db.create_content(g, 'More Group Content')
>>> db.list_persons() == [p]
True
>>> db.list_groups() == [g]
True
>>> db.list_content(g) == [c2, c3]
True
>>> db.unsubscribe(c3)
>>> db.list_content(g) == [c2]
True
>>> db.list_subscriptions() == [p, g, c, c2]
True

SOCIAL NETWORK

Person profiles are linked together via symmetric friendships to form
a social network.  Group profiles and person profiles are linked via
an asymmetric membership relationship.

    db.list_friends(person) -> list of person IIDs
    db.list_members(group) -> list of person IIDs
    db.list_memberships(person) -> list of group IIDs

>>> db.list_members(g) == [p]
True
>>> db.list_memberships(p) == [g]
True

These relationships are generally created when someone creates a
single-use invitation code and passes it to someone else, who accepts
it.  Invitation codes are easy to write down or type.  Each one must
be kept secret and should not be revealed to any third party.

    db.create_friendship_invitation_code() -> string
    db.create_membership_invitation_code(group, roles=[]) -> string

Anybody can create a friendship invitation code, but only an
administrator of a group can create a membership invitation code for
that group.  The person who creates a group is automatically an
administrator (and also has certain permanent rights as the owner of
the group).  A membership invitation can optionally specify a list of
named roles, granting additional authority to the member.  These can
include "admin", "author", and user-defined roles -- see Authorization
below.

Someone receiving an invitation code can destructively fetch the
corresponding invitation object from OpenDHT, subscribe to and view
the pertinent profile, consider the source of the code, and then, at
his discretion, accept the invitation:

    db.fetch_invitation(code) -> invitation instance
    invitation.iid
    invitation.accept(db)

Invitations that are not fetched within one week expire and leave no
trace in the system.

The roles assigned to a group member can be queried and updated (by an
administrator of the group) as follows:

    db.get_roles(group, person) -> list of strings
    db.set_roles(group, person, roles)

Friends and members can be removed as follows:

    db.revoke_friendship(person)
    db.revoke_membership(group, person)

A member of a group can cancel his membership as follows:

    db.cancel_membership(group)

DATASET CONTENTS

The contents of a dataset consists of key-value pairs, where keys are
paths and values are arbitrary byte strings.  A path encodes a
sequence of one or more non-empty strings by joining them together
using the slash character ('/') as a separator.  We can think of the
elements of a path other than the last as naming folders.

    db.get(iid, path) -> value
    db.put(iid, path, value)

>>> db.put(c, '/data/color', 'red')
>>> db.get(c, '/data/color')
'red'

Path elements are stored internally as UTF-8.  Alternatively, an
arbitrary non-empty byte string can be used as a path element by
encoding it in hexadecimal with a prefix of "0x".  Hex encoding is
naturally required for path elements that contain slashes or start
with "0x".

Although values are arbitrary byte strings, we provide convenience
methods for UTF-8 text values and JSON objects (JavaScript Object
Notation, http://json.org/).  It is also possible to fetch the size of
a value without fetching the value itself.

    db.get(iid, path, type=None) -> value
    db.get_text(iid, path) -> unicode
    db.get_json(iid, path) -> dict
    db.get_size(iid, path) -> integer

Specifying a type of 'text', 'json', or 'size' in the first method is
equivalent to calling one of the other three.  If the value is not
defined in the dataset, None is returned, except for get_size(), which
raises LookupError.

>>> db.get_text(c, '/data/color')
u'red'
>>> db.get_size(c, '/data/color')
3

Values passed into put() are encoded using UTF-8 or JSON if they are
unicode or dictionaries, respectively.  Byte strings are stored as-is.

    db.put(iid, path, value)

>>> db.put(c, '/data/info.json', {'dimensions': [2, 4]})
>>> db.get_json(c, '/data/info.json')
{u'dimensions': [2, 4]}

It is often appropriate to keep people from overwriting each other's
data.  For this purpose, we provide the post() method, which is like
put() except that it replaces any path element that consists of a
single exclamation point ("!") with the fingerprint of the
cryptographic public key used to sign the update.  The effect is as if
the path can have multiple values, posted by different people.
Intentional overwriting can be prevented by granting write permission
only for paths that include the signer's fingerprint in the
appropriate position -- see the discussion about authorization below.

    db.post(iid, path, value) -> expanded path

>>> path = db.post(c, '/data/comments/!/comment.txt', 'Cool!')
>>> db.get(c, path)
'Cool!'

It is possible to delete a value from the dataset:

    db.delete(iid, path)

This is accomplished in our distributed setting by creating an
invisible tombstone item for the path.

>>> db.delete(c, path)

One can query a dataset for all paths that begin with a particular
prefix and have at least one more element than the prefix:

    db.keys(iid, path_prefix='') -> list of relative paths

The prefix is removed from each of the returned paths.  The results
are returned in log order -- that is, according to when the current
value of the key was added to the database.

>>> db.keys(c, '/data')
[u'color', u'info.json']

The keys() method also enables monitoring a dataset for changes.  Each
time it is called, it saves the database's mark value (an integer) in
the DB instance.  This value can be fetched and passed into the next
query as last_mark to return only relative paths whose value has been
modified (set or deleted) since the previous query:

    db.keys(iid, path_prefix='', last_mark=0) -> list of relative paths
    db.get_last_mark() -> integer

For convenience, we provide a way to access the contents of a folder
as a Python mapping, where the keys are paths relative to the folder.

    db.get_view(iid, base_path='', type=None) -> looks like a dict

Under the hood, operations on the view are implemented using the get,
put, delete, and keys methods.

>>> v = db.get_view(c, '/data')
>>> v['color']
'red'
>>> v['color'] = 'blue'

AUTHORIZATION

Which of the above operations are permitted and which will raise an
exception ("Unauthorized") is a function of the imprinted persona and
its relationship to the domain in question.  In other words, it
depends on the user.

Adding a content dataset to somebody else's personal domain is not
permitted.  Adding a content dataset to a group is permitted only if
the user created the group or is an administrator or author -- that
is, a member with the role "admin" or "author".

The creator of a domain is permitted to manage the relationships
entered into by that domain.  In the case of a group domain, the group
administrators can also manage the membership of the group.  In fact,
the creator and administrators have unrestricted rights to write to
the domain's profile, where the information about the domain's
relationships is stored.

Unless the creator of a content dataset decides otherwise, he is the
only one who can write to the dataset.  The add_rights() method can be
used to permit others to do so:

    db.add_rights(iid, type, pattern, roles=None)
    db.clear_rights(iid, type, pattern)

There are currently two possibilities for the rights type:
"member-write" and "visitor-write".  The first grants write rights to
domain members, while the second grants the same rights to anyone and
everyone.  If a list of roles (strings) is specified with a rights
type of "member-write", the rights are granted only to members with at
least one of those roles.

The pattern determines which paths those people are permitted to
modify (set or delete).  A pattern is matched against a path
element-by-element.  "*" matches an arbitrary element.  "!" matches
the hexadecimal fingerprint of the cryptographic public key used to
sign the update.  "..." matches all the remaining elements of the
path, of which there must be at least one.  Any other string matches
only an identical element.

Because rights are stored in the dataset itself (in the folder
"/rights"), granting unrestricted rights to the dataset delegates
rights management rights.  For instance, the following allows domain
administrators to manage the dataset:

    db.add_rights(iid, 'member-write', '/...', roles=['admin'])

ENCRYPTION

Each domain has an associated encryption key that is available to all
members or friends:

    db.get_domain_cipher(domain) -> cipher

The cipher instance can be passed to the various data access methods
if access to the value is to be limited to members/friends:

    db.get(iid, path, type=None, cipher=None) -> value
    db.get_text(iid, path, cipher=None) -> unicode
    db.get_json(iid, path, cipher=None) -> dict

    db.put(iid, path, value, cipher=None)
    db.post(iid, path, value, cipher=None) -> expanded path
    db.get_view(iid, base_path='', type=None, cipher=None) -> looks like a dict

Encryption is primarily useful in personal and group profiles.
Content datasets are normally only available to domain members/friends.

ADDITIONAL JSON SUPPORT

The following methods allow for creation and updating of JSON objects
using convenient keyword argument syntax:

    db.put_json(iid, path, cipher=None, **kwargs)
    db.post_json(iid, path, cipher=None, **kwargs) -> expanded path
    db.update_json(iid, path, cipher=None, **kwargs)

>>> db.update_json(c, '/data/info.json', shape='rectangular')
>>> db.get(c, '/data/info.json')
'{"dimensions": [2, 4], "shape": "rectangular"}'

>>> db.close()
"""

from __future__ import with_statement
from contextlib import contextmanager

import os
import time
import hmac
from hashlib import sha1
from binascii import hexlify, unhexlify
import logging
import random

import simplejson
from apsw import Connection

from fusion.lib.cipher import Cipher
from fusion.lib.sexp import encode, decode
from fusion.core.identity import generate, decode_identity
from fusion.core.signature import decode_signature
from fusion.core.item import (
    hex_to_buffer, hex,
    item_from_row, row_from_item,
    create_root_item, create_kv_item)
from fusion.core.rights import verify, parse_json_object
from fusion.db.node_id import get_node_id
from fusion.db.view import View
from fusion.db.path import path_to_list, list_to_path, decode_text
from fusion.db.escrow import get_escrow
from fusion.db.invitation import (
    generate_code, strengthen, fetch_invitation,
    create_friendship_invitation, create_friendship_invitation_code,
    create_membership_invitation, create_membership_invitation_code)
from fusion.db.util import *
from fusion.db.exceptions import *

INFO_TABLE = """
create table if not exists info(
    k primary key,
    v);"""

DATASET_TABLE = """
create table if not exists datasets(
    num integer primary key autoincrement,
    type,
    iid unique,
    domain default null,
    size integer default 0,
    info);"""

ITEM_TABLE = """
create table if not exists items(
    num integer primary key autoincrement,
    hash unique,
    type,
    iid,
    key,
    value,
    seq integer,
    salt,
    public_key,
    sig,
    size integer,
    total_size integer);

create index if not exists items_by_num
on items(iid, num);

create index if not exists items_by_key
on items(iid, key, num);"""

def init_doctest():
    """Initialize variables referred to in the doctests."""
    global identity0, identity1, identity2, identity3
    identity0 = generate()
    identity1 = generate()
    identity2 = generate()
    identity3 = generate()

_undefined = object()

class DB(object):
    """
    Database Instance
    """
    def __init__(self, path=':memory:', create=True):
        path = os.path.normpath(os.path.expanduser(path))
        dir = os.path.dirname(path)
        if dir and not os.path.exists(dir):
            os.makedirs(dir)

        self.my_id = get_node_id(path)

        self.connection = Connection(path)
        self.connection.setbusytimeout(2 * 60 * 1000)

        self.nesting = 0
        self.cursor = self.connection.cursor()

        self.last_mark = None # Database mark at the time of the last query.

        self.temporary_signers = None

        # Default is FULL, which is supposedly slow.  However, the
        # following seems to cause problems where the database is left
        # locked.
        #self.execute("pragma synchronous = NORMAL")

        if create:
            # This pragma has no effect if I execute it in a transaction.
            self.execute("pragma auto_vacuum=1")

            # Using execute() here instead of operate() makes opening the
            # database twice as fast, and avoids locking it.
            self.execute(INFO_TABLE + DATASET_TABLE + ITEM_TABLE)

    def close(self):
        self.cursor.close()
        self.connection.close()

    #
    # Database info table
    #

    def get_local_info(self, key, default=_undefined):
        c = self.execute("select v from info where k = ?", (key,))
        try:
            value = get_single_value(c)
            return from_sqlite_string(value)
        except LookupError:
            if default != _undefined:
                return default
            else:
                raise

    def get_safety_color(self):
	try:
            color = self.get_local_info('color')
        except LookupError:

            # color weight calibration (%)

            rw = 30
            gw = 59 
            bw = 11

            assert rw + gw + bw == 100, 'total color weight exeeds 100'

            # limits

            ls = '#9aa' # light
            ds = '#99a' # dark

            l = int(ls[1], 16)*rw + int(ls[2], 16)*gw + int(ls[3], 16)*bw
            d = int(ds[1], 16)*rw + int(ds[2], 16)*gw + int(ds[3], 16)*bw

            colors = ['#%x%x%x' % (r,g,b)
                      for r in range(16)
                      for g in range(16)
                      for b in range(16)
                      if d < r*rw + g*gw + b*bw < l]
            color = random.choice(colors)

            self.set_local_info('color', color)

	return color

    def set_local_info(self, key, value):
        value = to_sqlite_string(value)
        self.operate("replace into info values (?, ?)", (key, value))

    def clear_local_info(self, key):
        self.operate("delete from info where k = ?", (key,))

    def _test_local_info(self):
        """
        >>> db = DB()
        >>> db.set_local_info('answer', 42)

        >>> db.set_local_info('color', 'red')
        >>> db.get_local_info('color')
        'red'
        >>> db.set_local_info('color', u'green')
        >>> db.get_local_info('color')
        u'green'
        >>> db.clear_local_info('color')
        >>> db.get_local_info('color')
        Traceback (most recent call last):
            ...
        LookupError

        >>> db.get_local_info('answer', 99)
        42
        >>> db.get_local_info('nonexistent', 99)
        99
        >>> db.close()
        """
        pass

    #
    # Secrets
    #

    def get_cipher(self, reference, purpose, master_secret=None):
        return Cipher(self.get_secret(reference, purpose, master_secret))

    def get_secret(self, reference, purpose, master_secret=None):
        reference = hex(reference)
        secret = master_secret or self.get_my_master_secret()
        return hmac.new(secret, reference + purpose, sha1).digest()

    def _test_secrets(self):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> other = buffer(os.urandom(20))
        >>> s1 = db.get_secret(me, 'this')
        >>> s2 = db.get_secret(me, 'that')
        >>> s3 = db.get_secret(other, 'this')
        >>> s4 = db.get_secret(other, 'that')
        >>> secrets = [s1, s2, s3, s4]
        >>> map(len, secrets)
        [20, 20, 20, 20]
        >>> len(set(secrets))   # distinct
        4

        >>> cipher = db.get_cipher(me, 'test')
        >>> ciphertext = cipher.encrypt('Testing...')
        >>> len(ciphertext)
        32
        >>> cipher.decrypt(ciphertext)
        'Testing...'
        >>> db.close()
        """
        pass

    #
    # Domain secret
    #

    def get_domain_cipher(self, domain):
        return Cipher(self.get_domain_secret(domain))

    def get_my_domain_secret(self, my_iid=None, my_master_secret=None):
        with self.transaction():
            my_iid = my_iid or self.get_my_iid()
            return self.get_domain_secret(my_iid, my_iid, my_master_secret)

    def get_domain_secret(self, domain, my_iid=None, my_master_secret=None):
        domain = hex(domain)
        with self.transaction():
            my_iid = my_iid or self.get_my_iid()
            my_master_secret = my_master_secret or self.get_my_master_secret()

            # 1. Remembered domain secret
            hmac_key = self.get_secret(my_iid, 'hmac', my_master_secret)
            token = hmac.new(hmac_key, domain, sha1).hexdigest()
            cipher = self.get_cipher(my_iid, 'encrypt domain secret',
                                     my_master_secret)
            key = ['admin', 'secret', token]
            secret = self.get(my_iid, key, cipher=cipher)
            if secret:
                return secret

            # 2. Returned domain secret
            if domain != my_iid:
                hmac_key = self.get_my_domain_secret(my_iid, my_master_secret)
                token = hmac.new(hmac_key, domain + my_iid, sha1).hexdigest()
                key = ['admin', 'secret', token]
                for fp, value in self.get_view(domain, key).items():
                    cipher = self.get_cipher(fp, 'encrypt domain secret',
                                             my_master_secret)
                    return cipher.decrypt(value)

            raise LookupError

    def remember_domain_secret(self, domain, secret):
        domain = hex(domain)
        with self.exclusive_transaction():
            my_iid = self.get_my_iid()
            hmac_key = self.get_secret(my_iid, 'hmac')
            token = hmac.new(hmac_key, domain, sha1).hexdigest()
            key = ['admin', 'secret', token]
            cipher = self.get_cipher(my_iid, 'encrypt domain secret')
            self.put(my_iid, key, secret, cipher=cipher)

    def boomerang_domain_secret(self, domain, secret, fingerprint):
        with self.exclusive_transaction():
            my_iid = self.get_my_iid()
            my_secret = self.get_my_domain_secret(my_iid)
            srs = self.get_secret(fingerprint, 'encrypt domain secret')
            self.return_domain_secret(domain, secret, fingerprint,
                                      my_iid, my_secret, srs)

    def return_domain_secret(self, domain, domain_secret, fingerprint,
                             person, person_secret, srs):
        # Store domain secret in domain for access by person.
        domain = hex(domain)
        person = hex(person)
        fingerprint = hex(fingerprint)
        assert domain != person

        token = hmac.new(person_secret, domain + person, sha1).hexdigest()
        key = ['admin', 'secret', token, fingerprint]
        self.put(domain, key, domain_secret, cipher=Cipher(srs))

    def _test_domain_secret(self):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group = db.create_group(_identity=identity1)
        >>> len(db.keys(me, '/admin/secret'))
        2

        >>> db2 = DB()
        >>> me2 = db2.create_persona(_identity=identity2)
        >>> group2 = db2.create_group(_identity=identity3)
        >>> len(db2.keys(me2, '/admin/secret'))
        2

        >>> len(db.get_my_domain_secret())
        20
        >>> len(db.get_domain_secret(group))
        20
        >>> db.get_domain_secret(me2)
        Traceback (most recent call last):
            ...
        LookupError
        >>> sync(db2, db)
        >>> db.get_domain_secret(me2)
        Traceback (most recent call last):
            ...
        LookupError
        >>> db.get_domain_secret(group2)
        Traceback (most recent call last):
            ...
        LookupError

        >>> s = db2.get_domain_secret(group2)
        >>> db.remember_domain_secret(group2, s)
        >>> db.get_domain_secret(group2) == s
        True
        >>> len(db.keys(me, '/admin/secret'))
        3

        >>> s = db.get_domain_secret(group)
        >>> s2 = db2.get_domain_secret(me2)
        >>> fp2 = hexlify(os.urandom(20))
        >>> srs = db2.get_secret(fp2, 'encrypt domain secret')
        >>> db.return_domain_secret(group, s, fp2, me2, s2, srs)
        >>> sync(db, db2)
        >>> db2.get_domain_secret(group) == s
        True
        >>> len(db.keys(group, '/admin/secret'))
        1

        >>> i = db._create_friendship_invitation()
        >>> i.accept(db2)
        True
        >>> sync(db2, db)
        >>> db.get_domain_secret(me) == db2.get_domain_secret(me)
        True
        >>> db.get_domain_secret(me2) == db2.get_domain_secret(me2)
        True

        >>> db.close()
        >>> db2.close()
        """
        pass

    #
    # Private keys
    #

    def get_private_keys(self, domain):
        return list(self.gen_private_keys(domain))

    def gen_private_keys(self, domain):
        domain = hex(domain)
        if self.temporary_signers is not None:
            for identity in self.temporary_signers:
                yield identity
            return

        # 1. Returned private keys
        with self.transaction():
            try:
                my_iid, my_master_secret = self.get_persona()
                my_domain_secret = self.get_my_domain_secret(
                    my_iid, my_master_secret)
            except LookupError:
                return

            token = hmac.new(my_domain_secret,
                             domain + my_iid, sha1).hexdigest()
            prefix = ['admin', 'private_key', token]
            items = self.get_view(domain, prefix).items()

        for fp, value in reversed(items):
            try:
                yield self.decrypt_private_key(value, fp, my_master_secret)
            except ValueError:
                pass

        # 2. Remembered private keys
        with self.transaction():
            hmac_key = self.get_secret(my_iid, 'hmac', my_master_secret)
            token = hmac.new(hmac_key, domain, sha1).hexdigest()
            prefix = ['admin', 'private_key', token]
            values = self.get_view(my_iid, prefix).values()

        for value in reversed(values):
            try:
                yield self.decrypt_private_key(value, my_iid, my_master_secret)
            except ValueError:
                pass

    def remember_private_key(self, domain, identity, visitor=False):
        if self.temporary_signers is not None:
            return

        domain = hex(domain)
        with self.exclusive_transaction():
            existing = self.get_private_keys(domain)
            if existing and visitor:
                return
            if identity.digest() in [x.digest() for x in existing]:
                return

            my_iid, my_master_secret = self.get_persona()
            ciphertext = self.encrypt_private_key(identity, my_iid,
                                                  my_master_secret)
            hmac_key = self.get_secret(my_iid, 'hmac', my_master_secret)
            token = hmac.new(hmac_key, domain, sha1).hexdigest()
            key = ['admin', 'private_key', token, sha1(ciphertext).hexdigest()]
            self.put(my_iid, key, ciphertext)

    def boomerang_private_key(self, domain, identity,
                              my_iid=None, my_domain_secret=None):
        fp = identity.fingerprint
        with self.exclusive_transaction():
            ciphertext = self.encrypt_private_key(identity)
            person = my_iid or self.get_my_iid()
            secret = my_domain_secret or self.get_my_domain_secret(my_iid)
            with self.override_signers(identity):
                self.return_private_key(domain, person, fp, ciphertext, secret)

    def return_private_key(self, domain, person, fingerprint,
                           ciphertext, person_secret):
        domain = hex(domain)
        person = hex(person)
        fingerprint = hex(fingerprint)

        token = hmac.new(person_secret, domain + person, sha1).hexdigest()
        key = ['admin', 'private_key', token, fingerprint]
        self.put(domain, key, ciphertext)

    def encrypt_private_key(self, identity, reference=None,
                            master_secret=None):
        purpose = 'encrypt private key'
        reference = reference or identity.fingerprint
        cipher = self.get_cipher(reference, purpose, master_secret)
        return cipher.encrypt(identity.encode_private())

    def decrypt_private_key(self, ciphertext, reference,
                            master_secret=None):
        purpose = 'encrypt private key'
        cipher = self.get_cipher(reference, purpose, master_secret)
        return decode_identity(cipher.decrypt(ciphertext))

    def _test_private_keys(self):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> domain = '1234abcd' * 5
        >>> len(db.get_private_keys(me))
        1
        >>> len(db.get_private_keys(domain))
        0
        >>> db.remember_private_key(domain, identity2, visitor=True)
        >>> len(db.get_private_keys(domain))
        1
        >>> db.remember_private_key(domain, identity2)
        >>> db.remember_private_key(domain, identity3, visitor=True)
        >>> len(db.get_private_keys(domain))
        1
        >>> db.remember_private_key(domain, identity3)
        >>> len(db.get_private_keys(domain))
        2
        >>> identities = db.gen_private_keys(domain)
        >>> [x.fingerprint for x in identities] == [
        ...     identity3.fingerprint, identity2.fingerprint]
        True

        >>> db2 = DB()
        >>> me2 = db2.create_persona(_identity=identity1)
        >>> group = db2.create_group(_identity=identity2)
        >>> sync(db2, db)
        >>> len(db.get_private_keys(group))
        0
        >>> fp = identity3.fingerprint
        >>> ciphertext = db.encrypt_private_key(identity3)
        >>> secret = db.get_my_domain_secret()
        >>> db2.return_private_key(group, me, fp, ciphertext, secret)
        >>> sync(db2, db)
        >>> [x.fingerprint for x in db.get_private_keys(group)] == [fp]
        True

        >>> db.add_rights(me, 'member-write', '/data/...')
        >>> db2.add_rights(me2, 'member-write', '/data/...')
        >>> sync(db, db2)
        >>> sync(db2, db)
        >>> db.put(me2, '/data/test', 'testing...')
        Traceback (most recent call last):
            ...
        Unauthorized
        >>> db2.put(me, '/data/test', 'testing...')
        Traceback (most recent call last):
            ...
        Unauthorized

        >>> i = db._create_friendship_invitation()
        >>> i.accept(db2)
        True
        >>> db2.put(me, '/data/test', 'testing...')
        >>> db2.get(me, '/data/test')
        'testing...'
        >>> sync(db2, db)
        >>> db.put(me2, '/data/test', 'testing...')
        >>> db.get(me2, '/data/test')
        'testing...'

        >>> db.close()
        >>> db2.close()
        """
        pass

    @contextmanager
    def override_signers(self, *identities):
        saved = self.temporary_signers
        self.temporary_signers = list(identities)
        try:
            yield
        finally:
            self.temporary_signers = saved

    #
    # My persona
    #

    def create_persona(self, name=u'', _identity=None):
        identity = _identity or generate()
        with self.exclusive_transaction():
            if self.get_local_info('my_iid', None):
                raise Unauthorized('Persona already initialized.')

            iid = self._create_domain(identity, 'person')
            self.set_persona(iid, os.urandom(20))

            secret = os.urandom(20)
            with self.override_signers(identity):
                self.remember_domain_secret(iid, secret)
            self.boomerang_private_key(iid, identity, iid, secret)

            self.add_rights(iid, 'member-write', '/admin/secret/*/!')
            self.add_rights(iid, 'member-write', '/admin/friendship/!/*')
            self.put(iid, '/public/name.txt', name)
            return iid

    def get_my_iid(self):
        return self.get_local_info('my_iid')

    def get_my_master_secret(self):
        return self.get_local_info('my_secret')

    def get_persona(self):
        with self.exclusive_transaction():
            return (self.get_my_iid(),
                    self.get_my_master_secret())

    def set_persona(self, iid, secret):
        iid = hex(iid)
        assert len(iid) == 40
        assert len(secret) == 20
        with self.exclusive_transaction():
            if self.get_local_info('my_iid', None):
                raise Unauthorized('Persona already initialized.')
            self.subscribe(iid)
            self.set_local_info('my_iid', iid)
            self.set_local_info('my_secret', secret)

    def clear_persona(self):
        with self.exclusive_transaction():
            self.clear_local_info('my_iid')
            self.clear_local_info('my_secret')

    def _test_persona(self):
        """
        >>> db = DB()

        >>> iid = db.create_persona(_identity=identity1)
        >>> db.get_type(iid)
        u'person'
        >>> db.get_domain(iid) == iid
        True
        >>> db.create_persona(_identity=identity2)
        Traceback (most recent call last):
            ...
        Unauthorized: Persona already initialized.

        >>> secret = db.get_my_master_secret()
        >>> db.get_my_iid() == iid
        True
        >>> db.get_persona() == (iid, secret)
        True

        >>> db.clear_persona()
        >>> db.get_persona()
        Traceback (most recent call last):
            ...
        LookupError
        >>> db.set_persona(iid, secret)
        >>> db.get_persona() == (iid, secret)
        True

        >>> db.close()
        """
        pass

    def reset_personal_code(self, passphrase):
        code = generate_code(20)
        key = strengthen(code + passphrase.encode('utf-8'), salt='-key-escrow')
        encryption_key = hmac.new(key, 'encryption key', sha1).digest()

        info = {'key': hexlify(key),
                'code': code,
                'passphrase': passphrase}
        n = len(simplejson.dumps(passphrase))
        if n < 100:
            info['padding'] = 'x' * (100 - n)

        with self.exclusive_transaction():
            iid, secret = self.get_persona()
            self.set_escrow_info(iid, info)
            self.put(iid, '/admin/escrow/master_secret', secret,
                     cipher=Cipher(encryption_key))

        return code

    def set_persona_from_code(self, code, passphrase):
        """
        Look up persona in OpenDHT.

        This may take tens of seconds, because it accesses OpenDHT.
        If it raises opendht.Error, the user should try again.  If it
        raises LookupError, no escrowed information was found.
        """
        key = strengthen(code + passphrase.encode('utf-8'), salt='-key-escrow')
        iid, secret = get_escrow(key)
        self.set_persona(iid, secret)

    def set_persona_from_iid_and_code(self, iid, code, passphrase):
        key = strengthen(code + passphrase.encode('utf-8'), salt='-key-escrow')
        encryption_key = hmac.new(key, 'encryption key', sha1).digest()
        with self.exclusive_transaction():
            secret = self.get(iid, '/admin/escrow/master_secret',
                              cipher=Cipher(encryption_key))
            if secret:
                self.set_persona(iid, secret)
            else:
                raise LookupError('No escrowed master secret in profile.')

    def get_escrow_info(self, iid):
        cipher=self.get_cipher(iid, 'encrypt escrow keys')
        return self.get_json(iid, '/admin/escrow/escrow_keys', cipher=cipher)

    def set_escrow_info(self, iid, info):
        cipher=self.get_cipher(iid, 'encrypt escrow keys')
        self.put(iid, '/admin/escrow/escrow_keys', info, cipher=cipher)

    def _test_personal_code(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity0)
        >>> code = db.reset_personal_code('This is a test...')

        >>> db2 = DB()
        >>> sync(db, db2)
        >>> db2.set_persona_from_iid_and_code(iid, code, 'This is a test...')

        >>> db.get_persona() == db2.get_persona()
        True
        >>> db2.get_escrow_info(iid)['passphrase']
        u'This is a test...'

        >>> db.close()
        >>> db2.close()
        """
        pass

    #
    # Groups
    #

    def create_group(self, name=u'', _identity=None):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group = db.create_group(_identity=identity1)
        >>> db.get_type(group)
        u'group'
        >>> db.get_domain(group) == group
        True
        >>> len(db.fingerprints(group))
        1
        >>> db.get_rights(group, 'member-write', '/...')
        {u'roles': [u'admin']}

        >>> db.add_fingerprint(group, identity2, dict(roles=['admin']))
        >>> with db.override_signers(identity2):
        ...     db.add_fingerprint(group, identity3)
        >>> len(db.fingerprints(group))
        3

        >>> db.list_members(group) == [me]
        True
        >>> db.list_memberships(me) == [group]
        True
        >>> db.close()
        """
        identity = _identity or generate()
        fp = identity.fingerprint
        with self.exclusive_transaction():
            my_iid = self.get_my_iid()
            iid = self._create_domain(identity, 'group')

            self.remember_domain_secret(iid, os.urandom(20))
            self.boomerang_private_key(iid, identity, my_iid)
            self.add_rights(iid, 'member-write', '/admin/secret/*/!')
            self.add_rights(iid, 'member-write', '/admin/membership/!/*')
            self.add_rights(iid, 'member-write', '/...', roles=['admin'])
            self.add_fingerprint(iid, identity, dict(roles=['admin']))

            shared_secret = os.urandom(20)
            self.affirm_membership(my_iid, None, iid, fp, shared_secret)
            self.affirm_membership(iid, fp, my_iid, None, shared_secret)

            self.put(iid, '/public/name.txt', name)
            return iid

    def _create_domain(self, identity, type):
        """
        >>> db = DB()
        >>> iid = db._create_domain(identity1, 'person')
        >>> info = db.get_info(iid)
        >>> info['size'] > 0
        True
        >>> info['type']
        u'person'
        >>> info['iid'] == info['domain'] == iid
        True
        >>> db.get_item_by_hash(iid).is_root()
        True

        >>> iid2 = db._create_domain(identity2, 'group')
        >>> info2 = db.get_info(iid2)
        >>> info2['type']
        u'group'

        >>> db._create_domain(identity3, 'hunos')
        Traceback (most recent call last):
            ...
        Unauthorized: Unknown dataset type.
        >>> db.close()
        """
        root = create_root_item(identity, type=type)
        iid = hexlify(root.iid)
        with self.exclusive_transaction():
            self.subscribe(iid)
            self.put_item(root)
            return iid

    #
    # Content
    #

    def create_content(self, domain, title=u''):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group = db.create_group(_identity=identity1)
        >>> iid1 = db.create_content(group)
        >>> db.get_type(iid1)
        u'content'
        >>> db.get_domain(iid1) == group
        True

        >>> db.add_fingerprint(group, identity2)
        >>> with db.override_signers(identity2, identity1):
        ...     iid2 = db.create_content(group)
        >>> db.get_owner(iid2).fingerprint == identity1.fingerprint
        True

        >>> db.add_fingerprint(group, identity3, dict(roles=['author']))
        >>> with db.override_signers(identity3, identity2, identity1):
        ...     iid3 = db.create_content(group)
        >>> db.get_owner(iid3).fingerprint == identity3.fingerprint
        True

        >>> db.close()
        """
        domain = hex(domain)
        with self.exclusive_transaction():
            for identity in self.gen_private_keys(domain):
                root = create_root_item(identity, type='content',
                                        parent=domain)
                if verify(self, root, check_subscription=False):
                    iid = hexlify(root.iid)
                    self.subscribe(iid)
                    self.put_item(root)
                    self.put(iid, '/info/title.txt', title)
                    return iid
            raise Unauthorized

    #
    # Dataset management
    #

    def get_type(self, iid):
        type1 = self._get_type1(iid)
        type2 = self._get_type2(iid)
        assert type1 == type2
        return type1

    def _get_type1(self, iid):
        iid = hex_to_buffer(iid)
        c = self.execute("select type from datasets"
                         " where iid = ?", (iid,))
        try:
            return get_single_value(c) # Can be None.
        except LookupError:
            return None

    def _get_type2(self, iid):
        info = self.get_json(iid, '')
        if info is None:
            return None
        return info['type']

    def get_domain(self, iid):
        domain1 = self._get_domain1(iid)
        domain2 = self._get_domain2(iid)
        assert domain1 == domain2
        return domain1

    def _get_domain1(self, iid):
        iid = hex_to_buffer(iid)
        c = self.execute("select type, domain from datasets"
                         " where iid = ?", (iid,))
        try:
            type, domain = get_single_row(c)
            return hexlify(domain or iid) if type else None
        except LookupError:
            return None

    def _get_domain2(self, iid):
        iid = hex(iid)
        info = self.get_json(iid, '')
        if info is None:
            return None
        return info.get('parent', iid)

    def get_owner(self, iid):
        item = self.get_item_by_hash(iid)
        if item is None:
            raise LookupError
        return item.get_identity()

    def list_persons(self):
        c = self.execute("select iid from datasets where type = 'person'")
        return map(hexlify, get_values(c))

    def list_groups(self):
        c = self.execute("select iid from datasets where type = 'group'")
        return map(hexlify, get_values(c))

    def list_content(self, domain):
        domain = hex_to_buffer(domain)
        c = self.execute("select iid from datasets"
                         " where domain = ?", (domain,))
        return map(hexlify, get_values(c))

    def list_subscriptions(self):
        c = self.execute("select iid from datasets")
        return map(hexlify, get_values(c))

    def list_iids_by_prefix(self, prefix):
        prefix = hex(prefix).upper()
        c = self.execute("select iid from datasets"
                         " where ? = substr(hex(iid), 1, ?)",
                         (prefix, len(prefix)))
        return map(hexlify, get_values(c))

    def _test_listing(self):
        """
        >>> db = DB()

        >>> me = db.create_persona(_identity=identity0)
        >>> group1 = db.create_group(_identity=identity1)
        >>> group2 = db.create_group(_identity=identity2)
        >>> group3 = '1234abcd' * 5; db.subscribe(group3)
        >>> group4 = '87654321' * 5
        >>> content1 = db.create_content(group1)
        >>> content2 = db.create_content(group2)
        >>> content3 = db.create_content(group2)

        >>> iids = db.list_subscriptions()
        >>> iids == [me, group1, group2, group3,
        ...          content1, content2, content3]
        True

        >>> db.list_iids_by_prefix(content2[:10]) == [content2]
        True
        >>> db.list_iids_by_prefix('') == db.list_subscriptions()
        True
        >>> db.list_iids_by_prefix(buffer(os.urandom(10))) == []
        True

        >>> db.list_persons() == [me]
        True
        >>> db.list_groups() == [group1, group2]
        True

        >>> db.list_content(group1) == [content1]
        True
        >>> db.list_content(group2) == [content2, content3]
        True
        >>> db.list_content(group3) == []
        True
        >>> db.list_content(group4) == []
        True

        >>> db.get_type(group4)
        >>> [db.get_type(iid) for iid in iids] == [
        ...     u'person', u'group', u'group', None,
        ...     u'content', u'content', u'content']
        True

        >>> db.get_domain(group4)
        >>> [db.get_domain(iid) for iid in iids] == [
        ...     me, group1, group2, None,
        ...     group1, group2, group2]
        True

        >>> db.get_owner(group1).fingerprint == identity1.fingerprint
        True
        >>> db.get_owner(group2).fingerprint == identity2.fingerprint
        True
        >>> db.get_owner(group3).fingerprint
        Traceback (most recent call last):
            ...
        LookupError

        >>> db.get_owner(content1).fingerprint == identity1.fingerprint
        True
        >>> db.get_owner(content2).fingerprint == identity2.fingerprint
        True
        >>> db.get_owner(content3).fingerprint == identity2.fingerprint
        True

        >>> db.close()
        """
        pass

    def subscribe(self, iid):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> iid1 = db.create_group(_identity=identity1)
        >>> iid2 = '1234abcd' * 5
        >>> db.subscribe(iid1)
        >>> db.subscribe(iid2)
        >>> db.subscribe(iid2)
        >>> db.list_subscriptions() == [me, iid1, iid2]
        True
        >>> info = db.get_info(iid2)
        >>> len(info['touched']) == len('yyyy-mm-ddThh:mm:ss')
        True
        >>> db.close()
        """
        iid = hex_to_buffer(iid)
        assert len(iid) == 20

        info = simplejson.dumps({'touched': get_timestamp()})
        try:
            self.operate("insert into datasets (iid, info)"
                         " values (?, ?)", (iid, info))
        except ConstraintError:
            pass                # Already there.

    def unsubscribe(self, iid):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group1 = db.create_group(_identity=identity1)
        >>> group2 = db.create_group(_identity=identity2)
        >>> content1 = db.create_content(group1)
        >>> content2 = db.create_content(group2)
        >>> db.list_subscriptions() == [me, group1, group2,
        ...                             content1, content2]
        True
        >>> db.unsubscribe(group1)
        >>> db.list_subscriptions() == [me, group2, content2]
        True
        >>> db.unsubscribe(content2)
        >>> db.list_subscriptions() == [me, group2]
        True
        >>> db.unsubscribe(me)
        Traceback (most recent call last):
            ...
        Unauthorized: Cannot unsubscribe to your own persona.
        >>> db.close()
        """
        iid = hex(iid)
        with self.exclusive_transaction():
            try:
                if self.get_my_iid() == iid:
                    raise Unauthorized('Cannot unsubscribe'
                                       ' to your own persona.')
            except LookupError:
                pass

            for additional in self.list_content(iid):
                self._remove_dataset(additional)

            self._remove_dataset(iid)

    def _remove_dataset(self, iid):
        iid = hex_to_buffer(iid)
        self.operate("delete from datasets where iid = ?;"
                     "delete from items where iid = ?;",
                     (iid, iid))

    def get_info(self, iid):
        """
        >>> db = DB()

        >>> domain = db.create_persona(_identity=identity1)
        >>> info = db.get_info(domain)
        >>> sorted(info.keys())
        [u'domain', u'iid', u'size', u'touched', u'type']
        >>> info['type']
        u'person'
        >>> info['iid'] == info['domain'] == domain
        True
        >>> info['size'] > 0
        True

        >>> content = db.create_content(domain)
        >>> info = db.get_info(content)
        >>> sorted(info.keys())
        [u'domain', u'iid', u'size', u'touched', u'type']
        >>> info['type']
        u'content'
        >>> info['iid'] == content
        True
        >>> info['domain'] == domain
        True
        >>> info['size'] > 0
        True

        >>> unknown = '1234abcd' * 5
        >>> db.subscribe(unknown)
        >>> info = db.get_info(unknown)
        >>> sorted(info.keys())
        [u'iid', u'size', u'touched']
        >>> info['iid'] == unknown
        True
        >>> info['size']
        0

        >>> db.close()
        """
        iid = hex_to_buffer(iid)
        c = self.execute("select type, domain, size, info from datasets"
                         " where iid = ?", (iid,))
        try:
            type, domain, size, info = get_single_row(c)
        except LookupError:
            return None

        info = simplejson.loads(info)
        info[u'size'] = size
        info[u'iid'] = hexlify(iid)
        if type:
            info[u'type'] = type
            info[u'domain'] = hexlify(domain or iid)

        return info

    def set_info(self, iid, info):
        # This simply has no effect if the subscription does not exist.
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group1 = db.create_group(_identity=identity1)
        >>> group2 = db.create_group(_identity=identity2)
        >>> unknown = '1234abcd' * 5
        >>> db.set_info(group1, {'size':-1, 'color': 'red'})
        >>> db.set_info(group2, {'size':-1, 'color': 'blue'})
        >>> db.set_info(unknown, {'size':-1, 'color': 'green'})
        >>> db.get_info(group1)['color']
        u'red'
        >>> db.get_info(group2)['color']
        u'blue'
        >>> db.get_info(group1)['size'] > 0
        True
        >>> db.get_info(group2)['size'] > 0
        True
        >>> db.get_info(unknown)
        >>> db.close()
        """
        iid = hex_to_buffer(iid)
        info.pop('size', None)
        info.pop('iid', None)
        info.pop('domain', None)
        info = simplejson.dumps(info, sort_keys=True)
        self.operate("update datasets set info = ?"
                     " where iid = ?", (info, iid))

    def update_info(self, iid, **kwargs):
        """
        >>> db = DB()

        >>> iid = db.create_persona(_identity=identity1)
        >>> db.update_info(iid, color='red', shape='round')
        >>> db.update_info(iid, shape='square', name='test')
        >>> info = db.get_info(iid)
        >>> info['color']
        u'red'
        >>> info['shape']
        u'square'
        >>> info['name']
        u'test'

        >>> unknown = '1234abcd' * 5
        >>> db.update_info(unknown, color='blue')
        >>> db.get_info(unknown) is None
        True

        >>> db.close()
        """
        with self.exclusive_transaction():
            info = self.get_info(iid)
            if info is not None:
                info.update(kwargs)
                self.set_info(iid, info)

    def touch(self, iid):
        """
        >>> db = DB()
        >>> iid = '1234abcd' * 5
        >>> db.subscribe(iid)
        >>> 'touched' in db.get_info(iid)
        True
        >>> db.set_info(iid, {})
        >>> 'touched' in db.get_info(iid)
        False
        >>> db.touch(iid)
        >>> info = db.get_info(iid)
        >>> len(info['touched']) == len('yyyy-mm-ddThh:mm:ss')
        True
        >>> db.close()
        """
        # XXX Use an in-memory hash table to throttle these to one per
        # minute per dataset.
        self.update_info(iid, touched=get_timestamp())

    def _set_dataset_type(self, iid, type, domain=None):
        iid = hex_to_buffer(iid)

        # XXX This part really belongs in fusion.core.rights.
        if type == 'content':
            try:
                domain = hex_to_buffer(domain)
            except Exception:
                raise Unauthorized
        elif type == 'person' or type == 'group':
            if domain is not None:
                raise Unauthorized
        else:
            raise Unauthorized('Unknown dataset type.')

        self.operate("update datasets set type = ?, domain = ?"
                     " where iid = ?", (type, domain, iid))

    def _update_dataset_size(self, iid, delta):
        iid = hex_to_buffer(iid)
        c = self.execute("select size from datasets"
                         " where iid = ?", (iid,))
        size = get_single_value(c) + delta
        self.operate("update datasets set size = ?"
                     " where iid = ?", (size, iid))

    #
    # Invitations
    #

    def create_friendship_invitation_code(self, **kwargs):
        return create_friendship_invitation_code(self, **kwargs)

    def create_membership_invitation_code(self, group, **kwargs):
        return create_membership_invitation_code(self, group, **kwargs)

    def fetch_invitation(self, code):
        return fetch_invitation(code)

    # The following are useful for testing without OpenDHT.

    def _create_friendship_invitation(self, **kwargs):
        return create_friendship_invitation(self, **kwargs)

    def _create_membership_invitation(self, group, **kwargs):
        return create_membership_invitation(self, group, **kwargs)

    #
    # Relationships
    #

    # XXX Moving the confirmation token into the value looks like a
    # good idea.  The value could be an s-expression of the form
    # [confirmation, encrypted-info].

    # XXX Try to simplify the interlocking relationship construction
    # somehow.  Consider moving the secrets into a private content
    # dataset, avoiding the use of the domain secret as an encryption
    # key, and distributing the domain secret using public-key
    # encryption.

    def list_friends(self, person, verify=True):
        details = self.list_friend_info(person)
        return self._list_relationships(details, verify)

    def list_members(self, group, verify=True):
        details = self.list_member_info(group)
        return self._list_relationships(details, verify)

    def list_memberships(self, person, verify=True):
        details = self.list_membership_info(person)
        return self._list_relationships(details, verify)

    def _list_relationships(self, details, verify):
        iids = [info['iid'] for info in details
                if not verify or info['verified'] and info['current']]
        return remove_duplicates(iids)

    def report_friends(self, person):
        """
        Get a list of verified friend IIDs and a count of unverified ones.
        """
        details = self.list_friend_info(person)
        return self._report_relationships(details)

    def report_members(self, group):
        """
        Get a list of verified member IIDs and a count of unverified ones.
        """
        details = self.list_member_info(group)
        return self._report_relationships(details)

    def report_memberships(self, person):
        """
        Get a list of verified group IIDs and a count of unverified ones.
        """
        details = self.list_membership_info(person)
        return self._report_relationships(details)

    def _report_relationships(self, details):
        verified = [info['iid'] for info in details
                    if info['verified'] and info['current']]
        unverified = [info['iid'] for info in details
                      if not info['verified']]

        verified = remove_duplicates(verified)
        unverified = remove_duplicates(unverified)

        unverified = [iid for iid in unverified
                      if self.get(iid, '') is None]

        return verified, len(unverified)

    def list_friend_info(self, person): # deprecated
        results = []
        with self.transaction():
            for fp in self.fingerprints(person):
                try:
                    info = self.get_friend_info(person, fp)
                    results.append(info)
                except LookupError:
                    continue
        return results

    def list_member_info(self, group):
        results = []
        with self.transaction():
            for fp in self.fingerprints(group):
                try:
                    info = self.get_member_info(group, fp)
                    results.append(info)
                except LookupError:
                    continue
        return results

    def list_membership_info(self, person):
        person = hex(person)
        results = []
        with self.transaction():
            try:
                cipher = self.get_domain_cipher(person)
            except LookupError:
                return []

            view = self.get_view(person, '/admin/membership',
                                 type='json', cipher=cipher)
            for info in view.values():
                group = hex(info['iid'])
                fp = hex(info['fingerprint'])
                shared_secret = unhexlify(info['secret'])
                verified = self.verify_membership(group, fp,
                                                  person, None,
                                                  shared_secret)
                info = {'iid': group, 'verified': verified}
                if verified:
                    info['current'] = fp in self.fingerprints(group)
                    info['name'] = self.get_text(group, '/public/name.txt')
                results.append(info)
            return results

    def get_friend_info(self, person, friend_fingerprint):
        person = hex(person)
        friend_fp = hex(friend_fingerprint)
        with self.transaction():
            cipher = self.get_domain_cipher(person)
            key = ['admin', 'friendship', friend_fp]
            view = self.get_view(person, key, type='json', cipher=cipher)
            for info in reversed(view.values()):
                friend = hex(info['iid'])
                person_fp = hex(info['fingerprint'])
                shared_secret = unhexlify(info['secret'])
                verified = self.verify_friendship(friend, person_fp,
                                                  person, friend_fp,
                                                  shared_secret)
                info = {'fingerprint': friend_fp,
                        'iid': friend, 'verified': verified}
                if verified:
                    info['current'] = (
                        friend_fp in self.fingerprints(person)
                        and person_fp in self.fingerprints(friend))
                    info['name'] = self.get_text(friend, '/public/name.txt')
                return info
            else:
                raise LookupError

    def get_member_info(self, group, fingerprint):
        group = hex(group)
        fingerprint = hex(fingerprint)
        with self.transaction():
            cipher = self.get_domain_cipher(group)
            key = ['admin', 'membership', fingerprint]
            view = self.get_view(group, key, type='json', cipher=cipher)
            for info in reversed(view.values()):
                iid = hex(info['iid'])
                shared_secret = unhexlify(info['secret'])
                verified = self.verify_membership(iid, None,
                                                  group, fingerprint,
                                                  shared_secret)
                info = {'fingerprint': fingerprint,
                        'iid': iid, 'verified': verified}
                if verified:
                    info['current'] = fingerprint in self.fingerprints(group)
                    info['name'] = self.get_text(iid, '/public/name.txt')
                return info
            else:
                raise LookupError

    def _test_friendships(self):
        """
        >>> db1 = DB()
        >>> db2 = DB()
        >>> db3 = DB()
        >>> p1 = db1.create_persona('Person 1', _identity=identity1)
        >>> p2 = db2.create_persona('Person 2', _identity=identity2)
        >>> p3 = db3.create_persona('Person 3', _identity=identity3)

        >>> i1 = db1._create_friendship_invitation()
        >>> sync(db1, db2)
        >>> i1.accept(db2)
        True

        >>> i2 = db2._create_friendship_invitation()
        >>> sync(db2, db3)
        >>> i2.accept(db3)
        True
        >>> sync(db3, db2)
        >>> sync(db2, db1, p1)
        >>> sync(db2, db1, p2)

        >>> db1.list_friends(p1) == [p2]
        True
        >>> db1.report_friends(p1) == ([p2], 0)
        True

        >>> db1.list_friends(p2) == [p1]
        True
        >>> db1.list_friends(p2, verify=False) == [p1, p3]
        True
        >>> db1.report_friends(p2) == ([p1], 1)
        True

        >>> db2.list_friends(p1) == [p2]
        True
        >>> db2.report_friends(p1) == ([p2], 0)
        True

        >>> db2.list_friends(p2) == [p1, p3]
        True
        >>> db2.report_friends(p2) == ([p1, p3], 0)
        True

        >>> db2.list_friends(p3) == [p2]
        True
        >>> db2.report_friends(p3) == ([p2], 0)
        True

        >>> for info in db1.list_friend_info(p2):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', True)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 1')
        ('verified', True)
        ----
        ('fingerprint', '...')
        ('iid', '...')
        ('verified', False)

        >>> info, = db1.list_friend_info(p1)
        >>> fp = info['fingerprint']
        >>> db1.remove_fingerprint(p1, fp)
        >>> sync(db1, db2)

        >>> db1.list_friends(p1) == []
        True
        >>> db1.report_friends(p1) == ([], 0)
        True

        >>> db1.list_friends(p2) == []
        True
        >>> db1.list_friends(p2, verify=False) == [p1, p3]
        True
        >>> db1.report_friends(p2) == ([], 1)
        True

        >>> db2.list_friends(p1) == []
        True
        >>> db2.report_friends(p1) == ([], 0)
        True

        >>> db2.list_friends(p2) == [p3]
        True
        >>> db2.list_friends(p2, verify=False) == [p1, p3]
        True
        >>> db2.report_friends(p2) == ([p3], 0)
        True

        >>> for info in db1.list_friend_info(p2):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', False)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 1')
        ('verified', True)
        ----
        ('fingerprint', '...')
        ('iid', '...')
        ('verified', False)

        >>> fp in db1.fingerprints(p1)
        False
        >>> info = db1.get_friend_info(p1, fp)
        >>> info['fingerprint'] == fp
        True
        >>> info['iid'] == p2
        True
        >>> for pair in sorted(info.items()):
        ...     print pair      #doctest: +ELLIPSIS
        ('current', False)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 2')
        ('verified', True)

        >>> db1.close()
        >>> db2.close()
        >>> db3.close()
        """
        pass

    def _test_memberships(self):
        """
        >>> db = DB()
        >>> me = db.create_persona('Person 1', _identity=identity0)
        >>> group = db.create_group('Group 1', _identity=identity1)

        >>> db2 = DB()
        >>> me2 = db2.create_persona('Person 2', _identity=identity2)
        >>> group2 = db2.create_group('Group 2', _identity=identity3)

        >>> invitation = db._create_membership_invitation(group)
        >>> invitation2 = db2._create_membership_invitation(group2)

        >>> sync(db, db2)
        >>> sync(db2, db)

        >>> invitation.group == group
        True
        >>> invitation.accept(db2)
        True

        >>> invitation2.group == group2
        True
        >>> invitation2.accept(db)
        True

        >>> db.list_members(group) == [me]
        True
        >>> db.report_members(group) == ([me], 0)
        True

        >>> db.list_members(group2) == [me2, me]
        True
        >>> db.report_members(group2) == ([me2, me], 0)
        True

        >>> db.list_memberships(me) == [group, group2]
        True
        >>> db.report_memberships(me) == ([group, group2], 0)
        True

        >>> sync(db2, db, group)
        >>> db.list_members(group) == [me]
        True
        >>> db.list_members(group, verify=False) == [me, me2]
        True
        >>> db.report_members(group) == ([me], 0) # Zero because we have data.
        True
        >>> for info in db.list_member_info(group):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', True)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 1')
        ('verified', True)
        ----
        ('fingerprint', '...')
        ('iid', '...')
        ('verified', False)

        >>> sync(db2, db)
        >>> db.list_members(group) == [me, me2]
        True
        >>> db.report_members(group) == ([me, me2], 0)
        True

        >>> for info in db.list_membership_info(me):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', True)
        ('iid', '...')
        ('name', u'Group 1')
        ('verified', True)
        ----
        ('current', True)
        ('iid', '...')
        ('name', u'Group 2')
        ('verified', True)

        >>> for info in db.list_member_info(group):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', True)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 1')
        ('verified', True)
        ----
        ('current', True)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 2')
        ('verified', True)

        >>> fp = invitation.fingerprint
        >>> db.remove_fingerprint(group, fp)
        >>> db.list_members(group) == [me]
        True
        >>> db.list_members(group, verify=False) == [me]
        True
        >>> db.report_members(group) == ([me], 0)
        True

        >>> sync(db, db2)
        >>> db2.list_memberships(me2) == [group2]
        True
        >>> db2.list_memberships(me2, verify=False) == [group2, group]
        True
        >>> db2.report_memberships(me2) == ([group2], 0) # We have data.
        True
        >>> for info in db2.list_membership_info(me2):
        ...     print '----'
        ...     for pair in sorted(info.items()):
        ...         print pair      #doctest: +ELLIPSIS
        ----
        ('current', True)
        ('iid', '...')
        ('name', u'Group 2')
        ('verified', True)
        ----
        ('current', False)
        ('iid', '...')
        ('name', u'Group 1')
        ('verified', True)

        >>> fp in db.fingerprints(group)
        False
        >>> info = db.get_member_info(group, fp)
        >>> info['fingerprint'] == fp
        True
        >>> info['iid'] == me2
        True
        >>> for pair in sorted(info.items()):
        ...     print pair      #doctest: +ELLIPSIS
        ('current', False)
        ('fingerprint', '...')
        ('iid', '...')
        ('name', u'Person 2')
        ('verified', True)

        >>> db.close()
        >>> db2.close()
        """
        pass

    def affirm_friendship(self, *args):
        return self._affirm_or_verify(relationship='friendship',
                                      action='affirm', *args)

    def verify_friendship(self, *args):
        return self._affirm_or_verify(relationship='friendship',
                                      action='verify', *args)

    def affirm_membership(self, *args):
        return self._affirm_or_verify(relationship='membership',
                                      action='affirm', *args)

    def verify_membership(self, *args):
        return self._affirm_or_verify(relationship='membership',
                                      action='verify', *args)

    def _affirm_or_verify(self, this_domain, this_fingerprint,
                          that_domain, that_fingerprint,
                          shared_secret, relationship, action):
        if relationship == 'friendship':
            assert this_fingerprint and that_fingerprint
        elif relationship == 'membership':
            assert this_fingerprint or that_fingerprint
        else:
            raise ValueError

        this_domain = hex(this_domain)
        that_domain = hex(that_domain)
        if this_fingerprint is not None:
            this_fingerprint = hex(this_fingerprint)
        if that_fingerprint is not None:
            that_fingerprint = hex(that_fingerprint)

        info = {'iid': that_domain,
                'secret': hexlify(shared_secret)}
        if that_fingerprint:
            info['fingerprint'] = that_fingerprint
        value = simplejson.dumps(info, sort_keys=True)
        confirmation = hmac.new(shared_secret, value, sha1).hexdigest()

        key = ['admin', relationship]
        if this_fingerprint:
            key.append(this_fingerprint)
        key.append(confirmation)

        if action == 'affirm':
            cipher = self.get_domain_cipher(this_domain)
            return self.post(this_domain, key, value, cipher=cipher)
        elif action == 'verify':
            return self.get(this_domain, key) is not None
        else:
            raise ValueError

    #
    # Canceling relationships
    #

    def revoke_friendship(self, person):
        with self.exclusive_transaction():
            my_iid = self.get_my_iid()
            self.revoke_relationship(my_iid, person)

    def revoke_membership(self, group, person):
        self.revoke_relationship(group, person)

    def cancel_membership(self, group):
        group = hex(group)
        with self.exclusive_transaction():
            my_iid = self.get_my_iid()
            cipher = self.get_domain_cipher(my_iid)
            view = self.get_view(my_iid, '/admin/membership',
                                 type='json', cipher=cipher)
            for path, info in view.items():
                if info['iid'] == group:
                    del view[path]

    def revoke_relationship(self, domain, person):
        domain = hex(domain)
        person = hex(person)
        with self.exclusive_transaction():
            fingerprints = self.list_fingerprints(domain, person)

            for fp in fingerprints:
                self.remove_fingerprint(domain, fp)

            view = self.get_view(domain, '/admin/secret')
            for path in view:
                if '/' in path:
                    token, fp = path.split('/', 1)
                    if fp in fingerprints:
                        del view[path]

    def list_fingerprints(self, domain, person):
        domain = hex(domain)
        person = hex(person)
        with self.transaction():
            type = self.get_type(domain)
            if type == 'person':
                relationship = 'friendship'
            elif type == 'group':
                relationship = 'membership'
            else:
                raise ValueError

            key = ['admin', relationship]
            cipher = self.get_domain_cipher(domain)
            view = self.get_view(domain, key, type='json', cipher=cipher)
            results = []
            for path, info in view.items():
                if info['iid'] == person:
                    fp, confirmation = path_to_list(path)
                    results.append(fp)
            return results

    def _test_revoke_friendship(self):
        """
        >>> db1 = DB()
        >>> db2 = DB()
        >>> p1 = db1.create_persona('Person 1', _identity=identity1)
        >>> p2 = db2.create_persona('Person 2', _identity=identity2)

        >>> invitation = db1._create_friendship_invitation()
        >>> sync(db1, db2)
        >>> invitation.accept(db2)
        True

        >>> sync(db2, db1)
        >>> db1.list_friends(p1) == [p2]
        True
        >>> len(db1.get_domain_secret(p2))
        20

        >>> db2.revoke_friendship(p1)

        >>> sync(db2, db1)
        >>> db1.list_friends(p1) == []
        True
        >>> len(db1.get_domain_secret(p2))
        Traceback (most recent call last):
            ...
        LookupError

        >>> len(db2.fingerprints(p1))
        1
        >>> len(db2.get_domain_secret(p1))
        20

        >>> db1.revoke_friendship(p2)
        >>> sync(db1, db2)

        >>> len(db2.fingerprints(p1))
        0
        >>> len(db2.get_domain_secret(p1))
        Traceback (most recent call last):
            ...
        LookupError

        >>> db1.close()
        >>> db2.close()
        """
        pass

    def _test_revoke_membership(self):
        """
        >>> db1 = DB()
        >>> p1 = db1.create_persona('Person 1', _identity=identity1)
        >>> g1 = db1.create_group('Group 1', _identity=identity2)

        >>> db2 = DB()
        >>> p2 = db2.create_persona('Person 2', _identity=identity3)

        >>> invitation = db1._create_membership_invitation(g1)
        >>> sync(db1, db2)
        >>> invitation.accept(db2)
        True

        >>> len(db2.fingerprints(g1))
        2
        >>> db2.list_members(g1) == [p1, p2]
        True
        >>> len(db2.get_domain_secret(g1))
        20

        >>> sync(db2, db1)
        >>> db1.revoke_membership(g1, p2)
        >>> sync(db1, db2)

        >>> db1.list_members(g1) == [p1]
        True
        >>> len(db1.get_domain_secret(g1))
        20

        >>> len(db2.fingerprints(g1))
        1
        >>> db2.list_members(g1) == []
        True
        >>> len(db2.get_domain_secret(g1))
        Traceback (most recent call last):
            ...
        LookupError

        >>> db2.list_memberships(p2) == []
        True
        >>> db2.list_memberships(p2, verify=False) == [g1]
        True
        >>> db2.cancel_membership(g1)
        >>> db2.list_memberships(p2, verify=False) == []
        True

        >>> db1.close()
        >>> db2.close()
        """
        pass

    def _test_cancel_membership(self):
        """
        >>> db1 = DB()
        >>> p1 = db1.create_persona('Person 1', _identity=identity1)
        >>> g1 = db1.create_group('Group 1', _identity=identity2)

        >>> db2 = DB()
        >>> p2 = db2.create_persona('Person 2', _identity=identity3)

        >>> invitation = db1._create_membership_invitation(g1)
        >>> sync(db1, db2)
        >>> invitation.accept(db2)
        True

        >>> db2.list_members(g1) == [p1, p2]
        True
        >>> db2.list_memberships(p2) == [g1]
        True

        >>> db2.cancel_membership(g1)

        >>> len(db2.fingerprints(g1))
        2
        >>> db2.list_members(g1) == [p1]
        True
        >>> db2.list_members(g1, verify=False) == [p1, p2]
        True
        >>> db2.list_memberships(p2) == []
        True
        >>> len(db2.get_domain_secret(g1))
        20

        >>> sync(db2, db1)
        >>> db1.revoke_membership(g1, p2)
        >>> sync(db1, db2)

        >>> len(db2.fingerprints(g1))
        1
        >>> db2.list_members(g1, verify=False) == []
        True
        >>> len(db2.get_domain_secret(g1))
        Traceback (most recent call last):
            ...
        LookupError

        >>> db1.close()
        >>> db2.close()
        """
        pass

    #
    # Roles
    #

    def get_roles(self, group, person):
        group = hex(group)
        person = hex(person)
        with self.transaction():
            roles = set()
            for fp in self.list_fingerprints(group, person):
                info = self.get_json(group, ['member', fp])
                if info and 'roles' in info:
                    roles.update(info['roles'])
            return sorted(roles)

    def set_roles(self, group, person, roles):
        group = hex(group)
        person = hex(person)
        assert isinstance(roles, (list, tuple, set))
        assert all(isinstance(role, (str, unicode)) for role in roles)
        with self.exclusive_transaction():
            roles = sorted(roles)
            for fp in self.list_fingerprints(group, person):
                info = self.get_json(group, ['member', fp])
                if info is not None:
                    if sorted(info.get('roles', [])) != roles:
                        self.update_fingerprint_info(group, fp, roles=roles)

    def _test_roles(self):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group = db.create_group(_identity=identity1)

        >>> db.get_roles(group, me)
        [u'admin']
        >>> db.set_roles(group, me, ['admin', 'developer'])
        >>> db.get_roles(group, me)
        [u'admin', u'developer']
        >>> db.set_roles(group, me, ['admin'])
        >>> db.get_roles(group, me)
        [u'admin']
        >>> db.set_roles(group, me, [])
        >>> db.get_roles(group, me)
        []

        >>> db.close()
        """
        pass

    #
    # Rights
    #

    def get_rights(self, iid, type, pattern):
        key = ['rights', type] + path_to_list(pattern)
        return self.get_json(iid, key)

    def add_rights(self, iid, type, pattern, **kwargs):
        key = ['rights', type] + path_to_list(pattern)
        self.put(iid, key, kwargs)

    def clear_rights(self, iid, type, pattern):
        key = ['rights', type] + path_to_list(pattern)
        self.delete(iid, key)

    def _test_rights(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> db.add_rights(iid, 'member-write', '/posts', roles=['author'])
        >>> db.add_rights(iid, 'member-write', '/comments')
        >>> db.get_rights(iid, 'member-write', '/posts')
        {u'roles': [u'author']}
        >>> db.get_rights(iid, 'member-write', '/comments')
        {}

        >>> db.clear_rights(iid, 'member-write', '/comments')
        >>> db.get_rights(iid, 'member-write', '/comments')

        >>> db.close()
        """
        pass

    #
    # Fingerprints
    #

    def fingerprints(self, domain):
        # if fingerprint in db.fingerprints(domain): ...
        # for fingerprint in db.fingerprints(domain): ...
        # for fingerprint, info in db.fingerprints(domain).items(): ...
        return self.get_view(domain, '/member', type='json')

    def get_fingerprint_info(self, domain, fingerprint):
        domain = hex(domain)
        fingerprint = hex(fingerprint)
        key = ['member', fingerprint]
        item = self.get_item(domain, key)
        if item is None or item.value is None:
            raise LookupError

        info = parse_json_object(item.value)
        info[u'identity'] = item.get_identity()
        info[u'fingerprint'] = unicode(fingerprint)

        return info

    def update_fingerprint_info(self, domain, fingerprint, **kwargs):
        assert 'iid' not in kwargs
        assert 'identity' not in kwargs
        with self.exclusive_transaction():
            info = self.get_fingerprint_info(domain, fingerprint)
            identity = info.pop('identity')
            info.pop('fingerprint')
            info.update(kwargs)
            self.add_fingerprint(domain, identity, info)

    def add_fingerprint(self, domain, identity, info={}):
        key = ['member', identity.fingerprint]
        self.put(domain, key, info, identity)

    def remove_fingerprint(self, domain, fingerprint):
        key = ['member', hex(fingerprint)]
        self.delete(domain, key)

    def _test_fingerprints(self):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity0)
        >>> group = db.create_group(_identity=identity1)

        >>> owner = db.get_owner(group)
        >>> info = db.get_fingerprint_info(group, owner.fingerprint)
        >>> sorted(info.keys())
        [u'fingerprint', u'identity', u'roles']
        >>> info['fingerprint'] == owner.fingerprint
        True
        >>> info['roles']
        [u'admin']

        >>> db.add_fingerprint(group, identity2, dict(name='Mickey'))
        >>> db.update_fingerprint_info(group, identity2.fingerprint,
        ...                            affiliation='Disney')
        >>> info = db.get_fingerprint_info(group, identity2.fingerprint)
        >>> info['name']
        u'Mickey'
        >>> info['affiliation']
        u'Disney'

        >>> db.update_fingerprint_info(group, identity1.fingerprint,
        ...                            name='Owner')
        >>> for fp in db.fingerprints(group):
        ...     info = db.get_fingerprint_info(group, fp)
        ...     print info['name']
        Mickey
        Owner

        >>> db.remove_fingerprint(group, identity2.fingerprint)
        >>> for info in db.fingerprints(group).values():
        ...     print info['name']
        Owner

        >>> db.get_fingerprint_info(group, identity2.fingerprint)
        Traceback (most recent call last):
            ...
        LookupError
        >>> db.get_fingerprint_info(group, identity3.fingerprint)
        Traceback (most recent call last):
            ...
        LookupError

        >>> identity1.fingerprint in db.fingerprints(group)
        True
        >>> identity2.fingerprint in db.fingerprints(group)
        False
        >>> identity3.fingerprint in db.fingerprints(group)
        False

        >>> db.close()
        """
        pass

    #
    # Fetch - insistently get.
    #

    def fetch(self, iid, path, type=None, cipher=None, default=None,
              autosubscribe=True, block=True,
              timeout=None, polling_interval=0.5):
        """
        >>> db = DB()
        >>> me = db.create_persona(_identity=identity1)
        >>> iid = db.create_group(_identity=identity2)
        >>> db.put(iid, 'answer', '42')
        >>> db.fetch(iid, 'answer')
        '42'
        >>> db.fetch(iid, 'nonexistent', block=False)
        >>> db.fetch(iid, 'nonexistent', timeout=0.1)
        >>> db.fetch(iid, 'nonexistent', timeout=0.1, polling_interval=0.01)
        >>> db.unsubscribe(iid)
        >>> db.list_subscriptions() == [me]
        True
        >>> db.fetch(iid, 'nonexistent', block=False)
        >>> db.list_subscriptions() == [me, iid]
        True
        >>> db.close()
        """
        if timeout is not None:
            assert timeout > 0
            deadline = time.time() + timeout

        while True:
            result = self.get(iid, path, type, cipher, default)
            if autosubscribe and result is None:
                self.subscribe(iid)
                autosubscribe = False

            if result is not None or not block:
                return result

            if timeout is not None:
                remaining = deadline - time.time()
                if remaining <= 0:
                    return None
                polling_interval = min(remaining, polling_interval)

            time.sleep(polling_interval)

    #
    # Application data access
    #

    def get(self, iid, path, type=None, cipher=None, default=None):
        assert type in (None, 'text', 'json', 'size')
        if type == 'size':
            return self.get_size(iid, path) # size of ciphertext

        iid = hex_to_buffer(iid)
        key = path_to_list(path)
        c = self.execute("select value from items"
                         " where iid = ? and key = ?"
                         " order by num desc",
                         (iid, buffer(encode(key))))
        try:
            v = get_single_value(c)
            if v is None:
                return default
            v = str(v)
        except LookupError:
            return default

        if cipher:
            v = cipher.decrypt(v)

        if type == 'text':
            return decode_text(v)
        elif type == 'json':
            return parse_json_object(v)
        else:
            return v

    def get_text(self, iid, path, cipher=None):
        return self.get(iid, path, type='text', cipher=cipher)

    def get_json(self, iid, path, cipher=None):
        return self.get(iid, path, type='json', cipher=cipher)

    def get_size(self, iid, path):
        iid = hex_to_buffer(iid)
        key = path_to_list(path)
        c = self.execute("select size from items"
                         " where iid = ? and key = ?"
                         " order by num desc",
                         (iid, buffer(encode(key))))
        try:
            size = get_single_value(c)
            if size is not None:
                return size
            raise KeyError
        except LookupError:
            raise KeyError

    def _test_get(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> db.put(iid, '/data/color', 'red')
        >>> db.put(iid, '/data/color', 'green')
        >>> db.get(iid, '/data/color')
        'green'
        >>> db.get(iid, '/data/color', type='size')
        5
        >>> db.delete(iid, '/data/color')
        >>> db.get(iid, '/data/color')
        >>> db.get(iid, '/data/color', type='size')
        Traceback (most recent call last):
            ...
        KeyError
        >>> db.get(iid, '/undefined')
        >>> db.get(iid, '/undefined', type='size')
        Traceback (most recent call last):
            ...
        KeyError

        >>> db.put(iid, '/data/color', 'green')
        >>> db.get_text(iid, '/data/color') == u'green'
        True
        >>> db.put(iid, '/data/color', 'vihre\xc3\xa4')
        >>> db.get_text(iid, '/data/color') == u'vihre\xe4'
        True
        >>> db.put(iid, '/data/color', '\xff')
        >>> db.get_text(iid, '/data/color') == u''
        True

        >>> db.put(iid, 'test', {'answer': 42})
        >>> db.get_json(iid, 'test')
        {u'answer': 42}
        >>> db.put(iid, 'test', '42')
        >>> db.get_json(iid, 'test')
        {}
        >>> db.put(iid, 'test', '{')
        >>> db.get_json(iid, 'test')
        {}

        >>> db.close()
        """
        pass

    def put(self, iid, path, value, reference_identity=None,
            expand_fingerprints=False, cipher=None):
        iid = hex(iid)
        key = path_to_list(path)

        if value is None:
            assert not ('!' in key and expand_fingerprints)
        elif isinstance(value, unicode):
            value = value.encode('utf-8')
        elif isinstance(value, dict):
            value = simplejson.dumps(value, sort_keys=True)
        else:
            assert isinstance(value, str)

        if cipher and value is not None:
            value = cipher.encrypt(value)

        with self.transaction():
            domain = self.get_domain(iid)
            if not domain:
                raise Unauthorized
            signers = self.get_private_keys(domain)

        if not signers:
            signing_identity = generate() # Do this with the database unlocked.
            with self.exclusive_transaction():
                reference_identity = reference_identity or signing_identity
                result = self._put(iid, key, value, signing_identity,
                                   reference_identity, expand_fingerprints)
                self.remember_private_key(domain, signing_identity,
                                          visitor=True)
                return result

        with self.exclusive_transaction():
            for signing_identity in signers:
                try:
                    return self._put(iid, key, value, signing_identity,
                                     reference_identity, expand_fingerprints)
                except Unauthorized:
                    continue

            signing_identity = signers[0]
            reference_identity = reference_identity or signing_identity
            return self._put(iid, key, value, signing_identity,
                             reference_identity, expand_fingerprints)

    def _put(self, iid, key, value, signer, reference, expand_fingerprints):
        if expand_fingerprints:
            key = [(signer.fingerprint if part == '!' else part)
                   for part in key]

        existing_item = self.get_item(iid, key)
        old_value = existing_item.value if existing_item else None
        if old_value != value:
            seq = existing_item.seq + 1 if existing_item else 1
            item = create_kv_item(iid, key, value, seq, signer, reference)
            self.put_item(item)

        if expand_fingerprints:
            return list_to_path(key)

    def post(self, iid, path, value, cipher=None):
        return self.put(iid, path, value, cipher=cipher,
                        expand_fingerprints=True)

    def delete(self, iid, path):
        self.put(iid, path, None)

    def _test_put(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> db.put(iid, '/data/text', u'testing...')
        >>> db.get(iid, '/data/text')
        'testing...'

        >>> db.put(iid, '/data/json', {'numbers': [1, 2, 3]})
        >>> db.get(iid, '/data/json')
        '{"numbers": [1, 2, 3]}'

        >>> db.put('1234abcd' * 5, '/hunos', None)
        Traceback (most recent call last):
            ...
        Unauthorized

        >>> with db.override_signers():
        ...     db.put(iid, 'dropbox/greeting', 'Hello!')
        Traceback (most recent call last):
            ...
        Unauthorized
        >>> db.add_rights(iid, 'visitor-write', '/dropbox/*')
        >>> with db.override_signers():
        ...     db.put(iid, 'dropbox/greeting', 'Hello!')
        >>> db.get(iid, 'dropbox/greeting')
        'Hello!'

        >>> with db.override_signers(identity3, identity2):
        ...     db.put(iid, 'test', 'No')
        Traceback (most recent call last):
            ...
        Unauthorized
        >>> with db.override_signers(identity3, identity2, identity1):
        ...     db.put(iid, 'test', 'Yes')
        >>> db.get(iid, 'test')
        'Yes'

        >>> db.put(iid, 'new', None)
        >>> db.get_item(iid, 'new')
        >>> db.put(iid, 'new', 'something')
        >>> db.get_item(iid, 'new').seq
        1
        >>> db.put(iid, 'new', 'something')
        >>> db.put(iid, 'new', None)
        >>> db.delete(iid, 'new')
        >>> db.get_item(iid, 'new').seq
        2

        >>> with db.override_signers(identity2):
        ...     path = db.post(iid, '/safe_dropbox/!/msg.txt', 'Hi!')
        Traceback (most recent call last):
            ...
        Unauthorized
        >>> db.add_rights(iid, 'visitor-write', '/safe_dropbox/!/*')
        >>> with db.override_signers(identity2):
        ...     path = db.post(iid, '/safe_dropbox/!/msg.txt', 'Hi!')
        >>> len(path) - len('safe_dropbox//msg.txt')
        40
        >>> db.get(iid, path)
        'Hi!'

        >>> db.delete(iid, path)
        >>> db.get(iid, path)

        >>> db.close()
        """
        pass

    def get_view(self, iid, base_path='', type=None, cipher=None):
        return View(self, iid, base_path, type, cipher)

    def _test_get_view(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> v = db.get_view(iid)
        >>> vv = db.get_view(iid, '/more')
        >>> vv_json = db.get_view(iid, '/more', type='json')
        >>> vv_text = db.get_view(iid, '/more', type='text')

        >>> v['1.txt'] = 'one'
        >>> v['/more/2.txt'] = 'two'
        >>> vv['34.json'] = {'values': [3, 4]}
        >>> v['/more/finnish/3.txt'] = u'kolme'
        >>> vv['/////finnish/4.txt'] = u'nelj\xe4'
        >>> vv['finnish/illegal.txt'] = '\xff'

        >>> v['/1.txt']
        'one'
        >>> vv['2.txt']
        'two'
        >>> v['/more/34.json']
        '{"values": [3, 4]}'

        >>> vv_json['34.json']
        {u'values': [3, 4]}
        >>> vv_json['2.txt']
        {}

        >>> vv_text['finnish/3.txt'] == u'kolme'
        True
        >>> vv_text['finnish/4.txt'] == u'nelj\xe4'
        True
        >>> vv_text['finnish/illegal.txt'] == u''
        True

        >>> del vv['finnish/3.txt']
        >>> del vv['nonexistent'] # In principle, this should raise KeyError.

        >>> 'finnish/3.txt' in vv
        False
        >>> '//finnish/4.txt//' in vv
        True
        >>> 'nonexistent' in vv
        False

        >>> vv.keys()
        [u'2.txt', u'34.json', u'finnish/4.txt', u'finnish/illegal.txt']
        >>> vv.items() == [
        ...     (u'2.txt', 'two'),
        ...     (u'34.json', '{"values": [3, 4]}'),
        ...     (u'finnish/4.txt', 'nelj\xc3\xa4'),
        ...     (u'finnish/illegal.txt', '\xff')]
        True
        >>> vv_json.items() == [
        ...     (u'2.txt', {}),
        ...     (u'34.json', {u'values': [3, 4]}),
        ...     (u'finnish/4.txt', {}),
        ...     (u'finnish/illegal.txt', {})]
        True
        >>> vv_text.items() == [
        ...     (u'2.txt', u'two'),
        ...     (u'34.json', u'{"values": [3, 4]}'),
        ...     (u'finnish/4.txt', u'nelj\xe4'),
        ...     (u'finnish/illegal.txt', u'')]
        True

        >>> v['/more'] = 'not included in vv'
        >>> v['/more/finnish'] = 'included in vv'
        >>> del v['/more/finnish/illegal.txt']
        >>> vv.keys()
        [u'2.txt', u'34.json', u'finnish/4.txt', u'finnish']

        >>> db.close()
        """
        pass

    #
    # JSON manipulation using keyword arguments
    #

    def put_json(self, iid, path, new_info=None, cipher=None, **kwargs):
        if new_info is None:
            new_info = kwargs
        else:
            new_info.update(kwargs)

        return self.put(iid, path, new_info, cipher=cipher)

    def post_json(self, iid, path, new_info=None, cipher=None, **kwargs):
        if new_info is None:
            new_info = kwargs
        else:
            new_info.update(kwargs)

        return self.post(iid, path, new_info, cipher=cipher)

    def update_json(self, iid, path, new_info=None, cipher=None, **kwargs):
        if new_info is None:
            new_info = kwargs
        else:
            new_info.update(kwargs)

        with self.exclusive_transaction():
            info = self.get_json(iid, path, cipher=cipher)
            if info is None:
                info = new_info
            else:
                info.update(new_info)

            self.put(iid, path, info, cipher=cipher)

    def _test_json(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> db.put_json(iid, 'x', {'color': 'red'}, shape='round')
        >>> sorted(db.get_json(iid, 'x').items())
        [(u'color', u'red'), (u'shape', u'round')]
        >>> db.put_json(iid, 'x', color='blue', answer=42)
        >>> sorted(db.get_json(iid, 'x').items())
        [(u'answer', 42), (u'color', u'blue')]

        >>> db.post_json(iid, 'y', {'color': 'red'}, shape='round')
        u'y'
        >>> sorted(db.get_json(iid, 'y').items())
        [(u'color', u'red'), (u'shape', u'round')]
        >>> db.post_json(iid, 'y', color='blue', answer=42)
        u'y'
        >>> sorted(db.get_json(iid, 'y').items())
        [(u'answer', 42), (u'color', u'blue')]

        >>> db.update_json(iid, 'z', {'color': 'red'}, shape='round')
        >>> sorted(db.get_json(iid, 'z').items())
        [(u'color', u'red'), (u'shape', u'round')]
        >>> db.update_json(iid, 'z', color='blue', answer=42)
        >>> sorted(db.get_json(iid, 'z').items())
        [(u'answer', 42), (u'color', u'blue'), (u'shape', u'round')]

        >>> db.close()
        """
        pass

    #
    # Queries
    #

    def keys(self, iid, path_prefix='', last_mark=0):
        key_prefix = path_to_list(path_prefix)

        query = "select num, key, size from items where iid = ?"
        params = [hex_to_buffer(iid)]

        prefix = ''
        if key_prefix:
            prefix = encode(key_prefix)[:-1]
            query += " and key > ?"
            params += [buffer(prefix)]

        if last_mark:
            query += " and num > ?"
            params += [last_mark]

        query += " order by key asc, num desc"

        results = []
        last_key = None
        with self.transaction():
            self.last_mark = self.get_current_mark()
            for num, key, size in self.execute(query, params):
                if key != last_key:
                    last_key = key

                    key = str(key)
                    if not key.startswith(prefix):
                        break           # Skip all the rest.

                    if size is not None or last_mark > 0:
                        result_key = decode(key)[len(key_prefix):]
                        if result_key:
                            results.append((num, result_key))

        results.sort()
        return [list_to_path(key) for (num, key) in results]

    # XXX Consider getting rid of get_last_mark() and providing two
    # versions of keys() instead, one of which returns multiple values.

    def get_last_mark(self):
        return self.last_mark

    def _test_keys(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)
        >>> iid2 = db.create_group(_identity=identity2)

        >>> db.put(iid, 'one', '1')
        >>> db.put(iid, 'two/alpha', '2a')
        >>> db.put(iid, 'two/beta', '2b')
        >>> db.put(iid, 'two/gamma', '2c')
        >>> db.put(iid, 'three', '3')
        >>> db.put(iid, 'two/beta', '')
        >>> db.delete(iid, 'two/gamma')
        >>> db.put(iid2, 'two/whatever', 'IRRELEVANT')

        >>> db.keys(iid)[-4:]
        [u'one', u'two/alpha', u'three', u'two/beta']
        >>> db.keys(iid, 'two')
        [u'alpha', u'beta']
        >>> db.keys(iid, 'nonexistent')
        []

        >>> db.keys(iid, 'two', last_mark=0)
        [u'alpha', u'beta']
        >>> last_mark = db.get_last_mark()

        >>> db.put(iid, 'two/gamma', '')
        >>> db.delete(iid, 'two/beta')
        >>> db.put(iid2, 'two/hunos', 'IRRELEVANT')
        >>> db.keys(iid, 'two', last_mark=last_mark)
        [u'gamma', u'beta']
        >>> db.keys(iid, 'two')
        [u'alpha', u'gamma']

        >>> db.keys(iid, 'rights/visitor-write')
        []
        >>> db.add_rights(iid, 'visitor-write', '/guestbook/...')
        >>> db.add_rights(iid, 'visitor-write', '/guestbook/...',
        ...               note='just testing...')
        >>> db.keys(iid, 'rights/visitor-write')
        [u'guestbook/...']
        >>> db.clear_rights(iid, 'visitor-write', '/guestbook/...')
        >>> db.keys(iid, 'rights/visitor-write')
        []

        >>> db.close()
        """
        pass

    def get_current_mark(self):
        """
        >>> db = DB()
        >>> db.get_current_mark()
        0
        >>> iid = db.create_persona(_identity=identity1)
        >>> db.get_current_mark()
        6
        >>> db.put(iid, 'test', '')
        >>> db.delete(iid, 'test')
        >>> db.get_current_mark()
        8
        >>> db.close()
        """
        c = self.execute("select seq from sqlite_sequence"
                         " where name='items'")
        try:
            return get_single_value(c)
        except LookupError:
            return 0

    #
    # Sync support
    #

    def gen_sync_agenda(self, iid, last_mark=0, current_mark=None):
        # XXX This sets self.last_mark, but the sync agent doesn't use it.

        query = "select num, hash, type from items where iid = ?"
        params = [hex_to_buffer(iid)]

        if last_mark:
            query += " and num > ?"
            params += [last_mark]

        if current_mark is not None:
            query += " and num <= ?"
            params += [current_mark]

        query += " order by num"

        hits = 0
        self.last_mark = self.get_current_mark()
        cursor = self.execute_with_new_cursor(query, params)
        try:
            for num, hash, type in cursor:
                hash = hexlify(hash)
                type = str(type)

                hits += 1
                if hits == 1:
                    self.last_mark = self.get_current_mark()

                depends = None
                if type == 'root' and self.get_type(hash) == 'content':
                    depends = self.get_domain(hash)

                yield num, hash, depends
        finally:
            cursor.close()

    def hash_to_num(self, hash):
        # Useful for quickly testing if we have a hash.
        hash = hex_to_buffer(hash)
        c = self.execute("select num from items"
                         " where hash = ?", (hash,))
        try:
            return get_single_value(c)
        except LookupError:
            return None

    def _test_sync_agenda(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)
        >>> iid2 = db.create_group(_identity=identity2)

        >>> [num for num, hash, depends in db.gen_sync_agenda(iid)]
        [1, 2, 3, 4, 5, 6, 8, 14]
        >>> [num for num, hash, depends in db.gen_sync_agenda(iid2)]
        [7, 9, 10, 11, 12, 13, 15, 16]
        >>> t1 = db.get_last_mark()
        >>> db.get_current_mark() == t1 == 16
        True

        >>> db.put(iid, 'data/one', '1')
        >>> db.put(iid, 'data/two', '2')
        >>> db.put(iid, 'data/three', '3')
        >>> db.put(iid2, 'data/whatever', 'IRRELEVANT')
        >>> db.put(iid, 'data/two', '2!')
        >>> db.delete(iid, 'data/three')

        >>> [(num, db.get_item_by_hash(hash).value)
        ...  for (num, hash, depends) in db.gen_sync_agenda(iid, last_mark=t1)]
        [(17, '1'), (21, '2!'), (22, None)]
        >>> t2 = db.get_last_mark()

        >>> hashes = [hash for (num, hash, depends)
        ...           in db.gen_sync_agenda(iid, last_mark=t1)]
        >>> [db.hash_to_num(hash) for hash in hashes]
        [17, 21, 22]

        >>> db.delete(iid, 'data/two')
        >>> db.put(iid, 'data/three', '3!')
        >>> [(num, db.get_item_by_hash(hash).value)
        ...  for (num, hash, depends) in db.gen_sync_agenda(iid, last_mark=t1,
        ...                                                 current_mark=t2)]
        [(17, '1')]
        >>> [(num, db.get_item_by_hash(hash).value)
        ...  for (num, hash, depends) in db.gen_sync_agenda(iid, last_mark=t2)]
        [(23, None), (24, '3!')]

        >>> [db.hash_to_num(hash) for hash in hashes]
        [17, None, None]

        >>> t3 = db.get_current_mark()
        >>> db.add_rights(iid, 'visitor-write', '/guestbook/...')
        >>> db.add_rights(iid, 'visitor-write', '/guestbook/...',
        ...               note='just testing...')
        >>> db.clear_rights(iid, 'visitor-write', '/guestbook/...')
        >>> [db.get_item_by_hash(hash).value
        ...  for num, hash, depends in db.gen_sync_agenda(iid, last_mark=t3)]
        ['{}', '{"note": "just testing..."}', None]

        >>> all(depends is None for _, _, depends in db.gen_sync_agenda(iid))
        True
        >>> all(depends is None for _, _, depends in db.gen_sync_agenda(iid2))
        True

        >>> iid3 = db.create_content(iid)
        >>> [depends is None for _, _, depends in db.gen_sync_agenda(iid3)]
        [False, True]
        >>> [depends == iid for _, _, depends in db.gen_sync_agenda(iid3)]
        [True, False]

        >>> db.close()
        """
        pass

    #
    # Raw Item access
    #

    def get_item_by_hash(self, hash):
        hash = hex_to_buffer(hash)
        c = self.execute("select * from items"
                         " where hash = ?", (hash,))
        try:
            row = get_single_row(c)
        except LookupError:
            return None

        return item_from_row(row)

    def get_item(self, iid, path):
        iid = hex_to_buffer(iid)
        key = path_to_list(path)
        c = self.execute("select * from items"
                         " where iid = ? and key = ?"
                         " order by num desc",
                         (iid, buffer(encode(key))))
        try:
            row = get_single_row(c)
        except LookupError:
            return None

        return item_from_row(row)

    def _test_get_item(self):
        """
        >>> db = DB()
        >>> iid = db.create_persona(_identity=identity1)

        >>> db.put(iid, 'test', 'testing...')
        >>> db.add_rights(iid, 'visitor-write', '/guestbook/...')
        >>> item1 = db.get_item(iid, 'test')
        >>> item2 = db.get_item(iid, 'rights/visitor-write/guestbook/...')
        >>> db.delete(iid, 'test')
        >>> db.clear_rights(iid, 'visitor-write', '/guestbook/...')
        >>> item3 = db.get_item(iid, 'test')
        >>> item4 = db.get_item(iid, 'rights/visitor-write/guestbook/...')

        >>> db.get_item_by_hash(item1.hash) is None
        True
        >>> db.get_item_by_hash(item2.hash) == item2
        True
        >>> db.get_item_by_hash(item3.hash) == item3
        True
        >>> db.get_item_by_hash(item4.hash) == item4
        True

        >>> db.close()
        """
        pass

    def put_item(self, item):
        with self.exclusive_transaction():
            if not verify(self, item):
                raise Unauthorized

            deleted = 0
            iid = item.iid
            key = item.key
            existing_item = self.get_item(iid, key)
            if existing_item is not None:
                if item == existing_item:
                    return existing_item.num # Already have it
                elif item < existing_item:
                    # If this is a rights or member item, we sort of
                    # actually do want it, even if we already have a
                    # newer version.  Oh well.
                    return None              # Already have a newer one.
                else:
                    # Remove older existing items, unless required for
                    # verification purposes.
                    if not key:
                        assert item.type == 'cancel'
                        deleted += self._purge("iid = ? and type = 'kv'",
                                               (iid,))
                    elif not item.is_member() and not item.is_rights():
                        deleted += self._purge("num = ?",
                                               (existing_item.num,))

            row = row_from_item(item)
            self.operate("insert into items"
                         " values (?,?,?,?,?,?,?,?,?,?,?,?)", row)
            num = self.connection.last_insert_rowid()

            if item.is_root():
                info = parse_json_object(item.value)
                type = info.get('type')
                domain = info.get('parent')
                self._set_dataset_type(iid, type, domain)

            inserted = row[-1]
            self._update_dataset_size(iid, inserted - deleted)

            return num

    def _purge(self, condition, parameters):
        c = self.execute("select total(total_size) from items where %s"
                         % condition, parameters)
        size = get_single_value(c)
        self.operate("delete from items where %s" % condition, parameters)
        return size

    def _test_put_item(self):
        """
        XXX
        """
        pass

    #
    # Database helpers
    #

    def exclusive_transaction(self):
        return self.transaction('exclusive')

    @contextmanager
    def transaction(self, strategy=None):
        self.begin(strategy)
        try:
            yield
        except:
            self._rollback()
            raise
        else:
            self.commit()

    def _rollback(self):
        try:
            self.rollback()
        except SQLError:
            logging.warning('Ignoring exception raised by rollback attempt.')

    def begin(self, strategy=None):
        if self.nesting == 0:
            if strategy is None:
                self.execute("begin")
            else:
                self.execute("begin %s" % strategy)
        self.nesting += 1

    def rollback(self):
        self.nesting -= 1
        if self.nesting == 0:
            self.execute("rollback")

    def commit(self):
        self.nesting -= 1
        if self.nesting == 0:
            self.execute("commit")

    def execute_with_new_cursor(self, statements, bindings=()):
        return self.execute(statements, bindings, new_cursor=True)

    def execute(self, statements, bindings=(), new_cursor=False):
        if new_cursor:
            cursor = self.connection.cursor()
        else:
            cursor = self.cursor

        while True:
            try:
                return cursor.execute(statements, bindings)
            except SchemaChangeError:
                continue

    def operate(self, statements, bindings=()):
        with self.exclusive_transaction():
            return self.execute(statements, bindings)

def _test():
    """
    The following are some overall tests from early on in development:

    >>> db = DB()

    >>> me = db.create_persona(_identity=identity1)
    >>> group = db.create_group(_identity=identity2)
    >>> content = db.create_content(me)
    >>> content2 = db.create_content(group)

    >>> db.list_persons() == [me]
    True
    >>> db.list_groups() == [group]
    True
    >>> db.list_content(me) == [content]
    True
    >>> db.list_content(group) == [content2]
    True
    >>> db.list_subscriptions() == [me, group, content, content2]
    True

    >>> view = db.get_view(content, 'data')
    >>> view['greeting'] = 'Hello world!'
    >>> view['greeting']
    'Hello world!'

    >>> full_view = db.get_view(content)
    >>> full_view['data/greeting']
    'Hello world!'

    >>> view['color'] = 'red'
    >>> view['color'] = 'blue'
    >>> view.keys()
    [u'greeting', u'color']
    >>> for k, v in view.iteritems():
    ...   print '%s: %s' % (k, v)
    greeting: Hello world!
    color: blue

    >>> del view['color']
    >>> view.keys()
    [u'greeting']

    >>> other_guy = identity3

    >>> with db.override_signers(other_guy):
    ...     db.put(content, 'data/shape', 'round')
    Traceback (most recent call last):
        ...
    Unauthorized
    >>> db.add_fingerprint(me, other_guy)
    >>> db.add_rights(content, 'member-write', '/data/...')
    >>> with db.override_signers(other_guy):
    ...     db.put(content, 'data/shape', 'square')
    >>> view.items()
    [(u'greeting', 'Hello world!'), (u'shape', 'square')]

    >>> db.add_fingerprint(group, other_guy)
    >>> with db.override_signers(other_guy):
    ...     db.create_content(group)
    Traceback (most recent call last):
        ...
    Unauthorized
    >>> db.update_fingerprint_info(group, other_guy.fingerprint, roles=['author'])
    >>> with db.override_signers(other_guy):
    ...     content3 = db.create_content(group)
    ...     db.add_rights(content3, 'member-write', '/comments/!/*')
    >>> db.put(content3, '/comments/wrong/something.txt', u'testing...')
    Traceback (most recent call last):
        ...
    Unauthorized
    >>> path = db.post(content3, '/comments/!/comment.txt', u'Cool!')
    >>> db.keys(content3) == [u'info/title.txt',
    ...                       u'rights/member-write/comments/!/*', path]
    True

    >>> db.post(content3, '/guestbook/!/entry.json', {})
    Traceback (most recent call last):
        ...
    Unauthorized
    >>> with db.override_signers(other_guy):
    ...     db.add_rights(content3, 'visitor-write', '/guestbook/!/entry.json')
    >>> path1 = db.post(content3, '/guestbook/!/entry.json', {'text': 'First!'})
    >>> with db.override_signers():
    ...     path2 = db.post(content3, '/guestbook/!/entry.json', {'text': 'Second!'})
    >>> db.get_view(content3, '/guestbook', type='json').values()
    [{u'text': u'First!'}, {u'text': u'Second!'}]

    >>> db.close()
    """
    pass

def sync(src, dst, iid=None):
    with dst.exclusive_transaction():
        with src.transaction():
            iids = [iid] if iid else src.list_subscriptions()
            for iid in iids:
                dst.subscribe(iid)
                for num, hash, depends in src.gen_sync_agenda(iid):
                    if depends:
                        sync(src, dst, depends)
                    item = src.get_item_by_hash(hash)
                    dst.put_item(item)

def remove_duplicates(seq):
    result_seq = []
    result_set = set()
    for x in seq:
        if x not in result_set:
            result_seq.append(x)
            result_set.add(x)
    return result_seq

if __name__ == '__main__':
    import doctest
    init_doctest()
    doctest.testmod()
